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DEVELOPMENT OF A STUDENT-PACED COURSE IN GENERAL PATHOLOGY 
UTILIZING A COMPUTER MANAGED EVALUATION SYSTEM 

Thomas H. Kent, Donald H. McClain and Stephen W. Wessels, Depart- 
ment of Pathology and University Computer Center,. University of 
Iowa 

The advantages, feasibility and problems associated with a student- 
paced course were investigated and computer managed evaluation system 
compared to paper and pencil testing mode. The development of a 
self-paced course was facilitated by explicit behavior objectives, 
a variety of learning materials referenced to the objectives and a 
large pool of test items. The advantages of the self-paced course, 
which was given in an otherwise fixed curriculum, included a moderate 
reduction in student completion time, more uniform mastery of the 
objectives, less conflict with other courses, a strongly positive 
student attitude, and more use of faculty for problem solving learn- 
ing experiences compared to information transfer. Problems In operat- 
ing the course were mainly associated with the administration, grad- 
ing and providing feedback on 2000 individual tests. Half of the 
tests were administered on cathrcde ray terminals utilizing Course- 
writer III language with disk storage of all student responses and 
scores . 

Major advantages of the computer administered tests over paper and 
pencil tests included immediate feedback to students, production of 
summaries of student performance and attitudes without manual mani- 
pulation of paper or figures, less worry about test security and less 
supervisory time required. Problems with computer administered tests 
included terminal and computer shutdowns and costs. By the end of 
the course students preferred the computer to paper and pencil tests. 
Scores and time were the same for the two cesting modes. 



FILMED FROM BEST AVAILABLE COPY 



1 



DEVELOPMENT OF A STUDENT-PACED COURSE IN GENERAL PATHOLOGY 
UTILIZING A COMPUTER MANAGED EVALUATION SYSTEM 

Thomas H. Kent, Donald H. McClaIn and Stephen W. Wessels, Depart- 
ment of Pathology* and University Computer Center, University 
of Iowa 

Two years ago at the RIME Conference the development of a General 
Pathology course based on specific behavioral objectives and a 
cricerlon referenced evaluation system was reported (1). The learning 
materials for the course had been developed to a degree that we 
felt many students could proceed through the course at a pace fast- 
er than the usual course schedule. The ultimate benefits of self- 
pacing would require flexibility in most parts of the curriculum 
and would allow many students to save some time in medical school 
and some students to reach higher levels of competence by spending 
more time than is presently allowed. The benefits of self-pacing a 
few courses is less, but does free some student time for other 
activities or pursuit of some topics in depth and serves to iron 
out competition between courses created by fixed examination times. 
The purpose of the present project was to determine feasibility, 
costs and problems associated with a self -paced course and to compare 
computer administered with paper and pencil tests. 

COURSE ORGANIZATION : The course is divided into three semi- 
Independent parts, each with its own learning objectives, learning 
materials and evaluation instruments. The Intent of the first part 
of the course is to make knowledge acquisition easy and efficient 
with a high level of mastery. This part of the course is divided into 
nine topic units and requires approximately two-thirds of the student's 
effort. All of the required information is available in written form 
with reference to the most appropriate source appended to each of 
the 100 objectives of this segment of the course. These written 
materials include a choice between two standard textbooks with page 
references for appropriate objectives, 13 programmed text units (2) 
covering three of the largest units in detail, and 21 handouts cover- 
ing parts of all units. In addition a series of 28 lectures is 
provided which suggests an average pace for the course and is design- 
ed to emphasize the importance of various topics, discuss frequency 
of various disease processes and their causes, present a few difficult 
topics (such as hemostatic disorders) and show gross photographs. 

The second segment of the course, called case analysis, requires 
approximately one fourth of the student's effort for the course and 
is directed toward correlating skills in microscopic observation with 
judgements on data presented in short typical case histories involv- 
ing 25 basic disease processes. The learning experience consists of 
48 histories with microscopic slides which the students practice 
analyzing in small groups with the assistance of an Instructor. 
The intent of this segment is the acquisition of problem-solving skills 
in situations that simulate those they will encounter in the practice 
of medicine. 
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The third segment, entitled introduction to the use of the labor- 
atories, comprises about one-twelfth of the student's effort and is 
designed to concern students with laboratory costs, quality control 
procedures, factors which affect laboratory test results, roles of 
laboratory personnel and availability of laboratory procedures in 
various types of hospitals. The learning experience consists of a 
tour of each of seven laboratory subdivisions and handouts to provide 
the appropriate factual information. We felt that our students pre- 
viously lacked sufficient orientation and knowledge of laboratory 
operations to make maximum use of them. 

COURSE EVALIJATION SYSTEM : The evaluation system is designed to 
fulfill two purposes: T) to provide feedback to the student on his 
progress and encourage him to perform at the mastery level and, 2) to 
provide feedback to the department concerning parts of the course 
that need improvement. The system is set up to provide formative 
(practice) and summative (final) tests for each segment of the course 
including each of the nine didactic units. The major problems in- 
volved w settvng up this system include: having enough test items 
specifically referenced to course objectives to provide for formative 
and alternate forms of summative tests, maintaining test security, 
administering a minimum of 2000 individual tests, and analyzing test 
and questionnaire data. 

The evaluation system for the case analysis segment of the course has 
been self-paced since its inception two years ago. A pool of test cases 
covering 25 basic disease processes in 11 organs is used to make up a 
series of 10 item tests. .The same sets of cases serve as formative 
tests as there is only a few points penalty for failure on the first 
try, and, in fact, about one-third of the students need to take the test 
more than once. 

The nine didactic unit exams, the final exam for the didactic units 
and the introduction to the use of the laboratories exam consist of 
multiple choice test items. We had planned to administer all of these 
tests on paper and pencil, but 5 weeks before the start of the course 
we found enthusiastic support from the University Computer Center and 
financial support from the Dean to develop a computer managed evaluation 
system for the multiple choice items. A lack of sufficient number of 
cathrode ray terminals (CRT's) during the first half of the course led^ 
us to assign the students on alternate units to paper and pencil 
testing mode and observe the differences in the two modes. 

The computer system utilized Coursewriter III language and an IBM 
360/65 computer. The computer was programmed to store on disk all 
student responses to test questions and questionnaire items and 
unit test scores. The program allowed the instructor to display 
on the terminals studerit test scores and individual responses to 
test questions. The questionnaire responses were programmed to remain 
anonymous v/hile still allowing the computer to correlate attitudinal 
data with performance data. 



The problem of sufficient test itenii was solved by our systematic 
collection of items with data from previous years, by a developing 
pathology test item pool among several schools by an informal organ- 
ization called Group for Research In Pathology Education (GRIPE), 
and by creation of a limited number of new questions. Three 
equivalent exams with items proportion to the objectives were developed 
for each of the 9 didactic units and the introduction to the lab- 
oratories segment. Four forms of the 32 item final didacitc exam 
were developed. Each item was assigned a minimum pass level as des- 
cribed by Taylor et al (3). Except for the final didactic exam, one of 
the exams served as the formative test and was given to the students 
with an answer key, minimum pass level and mastery level • The other 
two exams serve as alternate forms of the summative exam. 

The problem of test security was partially solved by the CRT administer- 
ed tests which prevented circulation of hard copy of tests and displayed 
the test item only when it was being answered. Initially we did not 
ive the student the answers to test items but provided a key phase 
mini objective) at the end of the test for each item missed. The 
students complained about not being able to see the correct answer, 
so we altered the computer program to display the correct answer 
after each student response. Security was maintained on paper and 
pencil test by having a limited number of test copies and by close 
supervision of the testing room, but we had no indication that the 
students were trying to beat the system. 

The problem of administering 2000 individual tests was a major one 
and required the hiring of an additional half-time test room supervisor. 
Hov/ever, this problem was greatly reduced in the last half of the course 
when we had more CRT's and students had learned how to operate the 
terminals without help. 

The problem of test grading and data analysis was different for 
the paper and pencil and CRT administered exams. The paper and pencil 
tests required hand grading and individually showing students the 
items they missed. At the end of the course the collective data was 
easily processed by our examination service but had to be translated 
to summary data sheets by hand. The CRT administered exams provided 
immediate feedback including the correct answer after each question, 
and, at the end of the test, the student's score, the minimum pass 
level, mastery level and key phrases for each item missed. Programming 
the computer to store and retrieve the student performance and question- 
naire data required considerable effort, but provided summaries that 
did not require further transfer of data. 

RESULTS : It would be difficult to say that students achieved at a 
higher level than in previous years, although the data in Table 1 
suggests that there was more uniform achievement of mastery in the 
self-paced unit oriented course of 1973 than in the paced 1972 course 
with midterm and final exams. 



Table 1. Percent of Students Achieving Mastery and Failing on 
First Try for 1972 and 1973 

% Mastery on First Try % Fail on First Try 



1972 1973 1972 1973 

Didactic Unit 1 72 93 0 0 

2 - 73 - 3 

3 59 33 2 8 

4 80 74 12 

5 51 83 6 2 

6 ?6 79 1 3 

7 70 80 13 

8 39 91 3 0 

9 90 78 0 0 
Intro to Labs 38 80 0 1 
Case Analysis - - 47 34 
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The week of the course in which students finished various segments 
and the whole course is shown in the figure. The students were 
explicitly told they had to finish in the allotted time for the course. 
One ftudent required an extra day. As indicated in the figure, 
approximately ]0% of the students finished 4 weeks early, 20^!!, 3 weeks 
early and 60%, 2 weeks early. 

The student attitude toward the course was strongly positive with 
all types of learning experiences and all instructors receiving an 
average or better rating. Programmed texts and case analysis dis- 
cussions rated the highest. Eighty-nine percent of the students pre- 
ferred self-paced unit testing compared to 2% for paced unit test- 
ing, 2% for self-paced midterms and finals, ]% for scheduled mid- 
term and final, and 6% not responding. 

The computer administered tests initially created a negative student 
response. The attitude became positive in the second half of the 
course when many problems involving terminals, programming, computer 
shutdowns, and familiarity with the system were resolved. At the 
end of the course 41% of students define cely preferred the computer 
mode, 42% said either mode was acceptable with half slightly favor- 
ing each mode, and 15% definitely preferred paper and pencil testing. 
Mean test scores for computer and paper and pencil administered tests 
were not statistically different and the time required to tal'.e the 
test was not different except for the first time on the computer 
terminal in which there was a pretest used to instruct students to 
use the terminal . 

Ninety-eight percent of the students indicated that they used the 
formative tests, 69% using them after completing studying of the 
unit, 19% using them before completing study of the unit and 10% us- 
ing them in a variable fashion. 

DISCUSSION : The major short term gain in this experimental student- 
paced course was a mild reduction in time required for students to 
complete the course, more uniformity in the degree of mastery of the 
various units, and a strongly positive student attitude toward self- 
pacing. The long-term gains are more difficult to substantiate ob- 
jectively. If students could be freed from the lock-step require- 
ments of other parts of the curriculum, the course could be run on 
a year around basis. Faculty effort for course preparation is pro- 
gressively reduced as part? of the course become more refined. This 
allowed redirection of faculty effort toward interaction with students 
in problem solving situations and toward ongoing course evaluation 
and critique. Two students who completed the course early gained 
additional experience in the surgical pathology laboratory. In the 
future more students will be given advanced opportunities. 

The cost of the course was somewhat increased over previous years. 



The amount of faculty effort was not changed, although more con- 
centrated with the course director who gave most of the lectures 
and developed the evaluation Instruments. Nearly the entire faculty, 
including visiting faculty, were Involved in the case analysis and 
introduction to the laboratories segment of the course. The computer 
administered evaluation system cost approximately $30 per student 
including programming, computer time, terminal rental and part-time 
secretarial help. This cost can be reduced by capital investment 
in terminals and finding more efficient means of sharing computer 
time; after debugging, programming and secretarial time can be re- 
duced to a minimum. We have no good estimate of previous costs in 
terms of faculty and secretarial time for preparation, administration 
and analysis of exams. The cost of secretarial time for individually 
administered paper and pencil tests is considerable. 

The Coursewriter III test administration system with student record 
storage can be interfaced with a computerized test item bank, so that, 
a course evaluation administration system can be created merely by 
specifying test item numbers and student identification numbers for 
the course. We feel that faculty effort can be reduced to decision 
making about appropriateness of individual test items, adding an 
occasional new item to suit new objectives, and evaluating computer 
prepared summaries of student and test item preformance. Secretarial 
time should be reduced to adding a few new test items, typing a series 
of numbers to define the tests for the course and being available in 
the testing center to solve problems that arise. 
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